Evaluation of Textual CBR Approaches

Evaluation is a crucial step in a research project, it demonstrates how well the chosen approach and the implemented techniques work, and can uncover limitations as well as point toward improvements and future research. A formal evaluation also facilitates comparing the project to previous work, and enables other researchers to assess its usefulness to their problems. Evaluating Textual CBR system is particularly challenging: The system will often comprise multiple aspects, e.g., interface, index-assignment, or case-based reasoning proper. The evaluation has to be carefully designed so that it disentangles the performance of those individual components which are to be the focus of the research project, or captures the performance of the overall system in solving a practical problem in industry. Textual CBR incorporates aspects of various disciplines which have different views on how to measure performance, and it is arguable which one of these measures is the best choice. In general, for Textual CBR, how to evaluate is still mostly an open issue (CBRIR 1998), and unfortunately, in some systems not even addressed at all. Here, we will discuss three major issues with respect to the evaluation of Textual CBR systems: